Least Squares Optimization

The following is a brief review of least squares optimization and constrained optimization techniques. I assume the reader is familiar with basic linear algebra, including the Singular Value decomposition (as reviewed in my handout Geometric Review of Linear Algebra). Least squares (LS) problems are those in which the objective function may be expressed as a sum of squares. Such problems have a natural relationship to distances in Euclidean geometry, and the solutions may be computed analytically using the tools of linear algebra. 1 Regression Least Squares regression is the most basic form of LS optimization problem. Suppose you have a set of measurements, y n gathered for different parameter values, x n. The LS regression problem is to find: min p N n=1 (y n − px n) 2 We rewrite the expression in terms of column N-vectors as: min p || y − pp x|| 2 Now we describe three ways of obtaining the solution. The traditional (non-linear-algebra) approach is to use calculus. If we set the derivative of the expression with respect to p equal to zero and solve for p, we get: p opt = y T x x T x. Technically, one should verify that this is a minimum (and not a maximum or saddle point) of the expression. But since the expression is a sum of squares, we know the solution must be a